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ABSTRACT 

Tlie purpose of this study iras to detersine irhich 
types of feedback had the sost influence on the revisions of 
instructional Materials performed by irriters in an individualized 
science cnrricnlus developtent project* Each of the individualized 
Science Instructional Systes (ISIS) project staff eesbers^ irhose job 
iras to revise siniconrse drafts produced and field*tested by the ISIS 
project^ participated in an evaluation of the types of feedback that 
say have influenced tlieir revisions of ISIS sinicourses* They 
rapk^ordered a list of types of feedback according to the asount of 
influence the feedback had on their actual revision of sinicourse 
materials* Verbal (audiotaped) feedback fros teachers iras rated as 
the most influential by nearly all of the subjects* Feedback. that 
involved information obtained from students irho used the 
instructional materials received the loirest ratings* Informal and 
verbal (l*e*^ nonquantitative) feedback iras rated as being more 
influential than feedback that iras more formal and less verbal (i*e*^ 
quantitative) * (Autlior/HLH) 
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Educational researchers involved in the formative eval* 
uation of instructional materials are typically faced with 
the problem of trying to collect information or provide 
experiences that will be relevant to those persons who will 
use the information or experiences to help them improve 
instructional products* It is not surprising, then, that 
this problem had to be faced by the evaluation staff of the 
Individualized Science Instructional System (ISIS)* Our 
staff established several formative evaluation procedures 
which were designed to provide revisors of the instructional 
materials with experiences and information that would have a 
desirable influence on their revision of ISIS materials 

It was not, however, until these formative evaluation 
procedures were put into operation that the following ques* 
tion could be asked: **Vhiah types of feedback had the meet* 
influence on the revisors of these inatruational materiale?** 
The remainder of this paper will describe a procedure that 



was used to answer the above question., it will also describe 
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the several types of feedback that were made available to 
these instructional material developers* 

A brief description of the ISIS Project is given below* 
The reading of this description may omitted by those per- 
sons who already are feuniliar with ISIS* The description, 
written by the Project Director, Dr. Ernest Burlcman, is taken 
from the ^'FOREWORD*' of each ISIS module* 



Evidence has been mounting that something is-^&lssiTtg from second* 
ary science teaching. More and nore» students are rejecting science 
courses and turning to subjects that they consider to be more prac* 
tical or significant. Numerous high school science teachers have 
concluded that what they are now teaching is appropriate for only a 
limited number of their students. 

As their concern has mounted* many science teachers have tried to 
find instructional materials that encompass more appropriate content 
and that allow them to work individually with students who have dif** 
ferent needs and talents. For the most part, this search has been 
frustrating because presently such materials are difficulty if not 
ln^ssible^ to find. 

The Individualized Science Instructional System (ISIS) project 
was organized to produce an alternative for those teachers who are 
dissatisifed with current secondary science textbooks. Consequent- 
ly* the content of the ISIS materials is uncoitventional as is the 
individualized teaching method that is built into them. In contrast 
with many current science texts which aim to "cover science**, ISIS 
has tried to be selective and to limit our coverage to the topics 
that we judge will be most useful to today*s students. 

Obviously the needs and problems of individual schools and students 
vsry widely. 'o accommodate the differences, ISIS decided against^ 
producing tightly structured^ pre-sequenced textbooks. Instead^ we 
are generating shorty self-contained modules that cover a wide range 
of topics. The modules csn be clustered into many types of courses^ 
and we hope that teachers and administrators will utilize this flex- 
ibility to tailor^-make curricula that are responsive to local needs 
and conditions. 

ISIS is a cooperative effort involving many individuals and agen* 
cies. More than 75 scjentists and educators have helped to generate 
the materials » and hundreds of teachers and thousands of students 
hsve been involved in the project *s nation-wide testing program. 
All of the ISIS endeavors have been supported by generous grants 
from the National Science Foundation. We hope that ISIS users will 
conclude that these large investments of time^ money, and effort 
have been worthwhile. 
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FEEDBACK PROCEDURES 

During the, 1974^1975 school year various feedback pro^ 
cedures were employed by the evaluation staff of the JSJS 
Project* Several of these procedures involved approximately 
50 schools, 100 teachers, and 6000 students who participated 
in the evaluation of 24 trial^draft edition ISIS minicourses* 
Other feedback procedures involved only project staff members 
or outside consultants* in all, ten major feedback procedures 
were identified and evaluated in this study* Each of these 
procedures is described, briefly, below* 

1* The revisors' visits to classes using the trial 

materials* 

Minlcourse writer-revisors visited a limited number 
of classrooms that were using materials developed by 
the ISIS staff* They observed teachers and students 
using these instructional materials and discussed their 
good and bad points with them* 
2* Reviews by "content experts " * 

The writer**revisors were provided with content 
reviews of the materials that were made by experts in 
the field who were not associated with the ISIS Project* 
Often the revisors would consult with the reviewer (by 
phone) to insure that the content of any revised mater- 
ial was scientifically accurate and up to date. 
3* Suggestions from other writer-revisors * 

ISIS writer-revisors did not work in isolation* 
Numerous formal and social meetings gave revisors an 
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opportunity to exchange ideas about materials they were 

working on as well as to offer suggestions that might 

benefit others in their revision tasks* 

4 * Suggestions from the Project Director or Associate 

Directors * 

This is a type of feedback that cannot be ignored 
in a large scale project such as ISIS* The revisors ^ 
frequently received both verbal and written instructions 
and suggestions from these Directors* 
5* Audio*taped comments from teachers who used the 
trial materials* 

ISIS has eight trial centers located throughout the 
country* On a regular basis teachers within each center 
gather for a meeting, part of which is devoted to the 
production of audio^tapes of discussions about minicourses 
they have tried* These feedback tapes are then forwarded 
to ISIS Project headquarters where they are listened to 
by the appropriate minicourse revisors. 
6* Comments written by teachers in trial edition book * 
lets* 

Revisors were provided with a compilation of page by 
page comments written by teachers directly on the mater*^ 
ials they had tried* These annotated pages pointed out 
pictures, figures, words, sentences or whole sections 
of instruction that caused unnecessary problems for stu** 
dents* Teachers also wrote on these pages suggestions 
for improvement of the materials* 
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7 . Teachers* written responses to questionnaire items . 
For each minicourse, teachers were asked to write 

responses to about ten open-ended questions that were 
designed to obtain feedback about specific items in 
that particular minicourse. These questions covered 
such things as specific problems with experiments, in- 
structional games or audio<-tapes that were part of the 
minicourse, the items of the minicourse test, the 
required activities in the minicourse, and the types of 
students who benefitted most or least from the mini- 
course. Written responses by all teachers were typed, 
compiled, and provided to revisors. 

8 . Students' responses to feedback questionnaire items . 
Each student who used ISIS trial version minicourses 

provided a detailed record of what he or she did in a 
particular minicourse and also recorded his or her atti- 
tudes toward each activity within the minicourse. The 
students recorded their data in response to daily multi- 
ple choice questions on machine scorable sheets which 
\tere read and conputer processed. This procedure enabled 
each revisor to be provided with an activity by activity 
breakdown of a minicourse in terms of the percentage of 
students who did the activity, how many found the direc* 
tions confusing, how many had difficulties with materials 
or equipment or the reading level, how much of each acti- 
vity students did, how much time students spent on an 
activity, whose help students needed to get through the 
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activity, and how interesting students found the acti- 
vity* In addition, the revisor was provided with data 
regarding students* overall ratings and reactions to 
the minicourse as well as background information on the 
Students who provided this feedback data* 
9* Students' test results * 

A minicourse revisor was provided with several ana* 
lyses of students' performance on the minicourse test* 
Pretest and posttest item analyses were provided for all 
test items covering the ^^Core** section of the minicourse* 
The ISIS Instructional Model holds all students respons** 
ible for mastering the Core Objectives in a minicourse* 
Although students are allowed to skip any Core Activities 
for which they have already mastered the corresponding 
objective, they are required to answer all Core test 
items* Therefore, the revisor was provided with data 
comparing the performance on each test item of students 
who did the Activity with the performance of those stu~ 
dents who skipped the Activity* Ideally, students who 
justifiably skip a Core Activity should do as well on 
the corresponding test item as those students who did the 
Activity* 

10* The revisors' examination of copies of booklets 
used by students * 

A revisor was also provided with a random sample of 
minicourse booklets used by students in each ISIS trial 
classroom* Since students wrote out their responses to 



text-embedded questions or problems directly in the 
trial edition booklets f an examination of these respon- 
ses allowed the revisor to pinpoint areas of instruction 
that were not being understood by students. Many students 
also wrote comments in their books des'jribing their likes 
and dislikes about the minicourse booklet. 

METHOD 

Each of seven ISIS project staff members whose job it 
was in the summer of 1975 to revise minicourse drafts produced 
and field-tested during the previous school year participated 
in an evaluation of various types of information or experiences 
that may have influenced their revisions of ISIS minicourses. 
All seven revisors were experienced developers of instructional 
materials and all held advanced academic degrees f mostly doc- 
torates. Each subject was given a randomly ordered list of ton 
different types of feedback that they had made use of (see 
Appendix A). Independently f subjects rank ordered the ten 
items on the list to show the comparative influence that the 
items had on their actual revisions of minicourse materials. 

RESULTS AND DISCUSSION 

The rank order that each of the seven revisors gave to 
the ten items (A-J) is shown in Table 1. 
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TABLE 1 

Rank Orders of Influence Given by Seven Revisors 
to Ten Feedback Procedures* 



Re vis or: 
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32 


12 


53 


51 


US 


I|2 


21 


51 


35 



*See, Appendix A for a listing of procedures; A-J. 

To determine the extent to which these seven rank orders 
tended to agree, Kendall's statistic, the coefficient of con- 
cordance, was computed.^ if the revisors were in perfect agree- 
ment in their rank orders, then the variance of the rank sums 
would be equal to the maximum possible variance of the rank 
stuns and the coefficient ivould equal 1.0. The coefficient^ was _ 
found to be 0.41. This did not indicate a high degree of con- 
cordance among the revisors since the variance of the rank sums 
was only 41% of the maximum possible variance. 



1. William L. Hays & Robert L. Winkler- Statistics: Probability, 
Infex^enoe, and Deoisicn. New Yorkt Holt, Rinehart, and Winston, 1971. 



The resultant low coefficient^ no doub^rwas due to the 
fact that thA revisers could not agree on the rank ordering 
of the four least influential itemsi and I* These 

four items had rank sums with a range of only 5 (see Table 1)* 
If each revisor had given exactly the same rank to each of 
these four least influential Items, the range of the rank sums 
would have been 21 (i*e*, a rank sum of 49 for 7 seventh place 
votes and a rank sum of 70 for 7 tenth place votes) and the 
variance between rank sums would, likewise, have been much 
greater* 

Despite the low coefficient of concordance, it is obvious 
from Table 1 that certain feedback procedures exerted consid- 
erably greater influence upon these seven revisors than other 
procedures* Five of the seven revisors ranked item C, the 
audio^taped comments from teachers, as the most influential 
of these ten feedback procedures* Item H, the comments by 
teachers in the minicourse booklets, also was highly influen- 
tial since all revisors ranked it among their top four choices* 
Five of the seven revisors ranked item J, the teachers* written 
responses to questionnaire items, to be among their top five 
choices* Note that each of these three high ranking items 
involved procedures that involved feedback from teachers Crather 
than from students) and that each of these three procedures 
could be categorized^ as '^^j}^, verbal and ijiforinal_ (i*e* # non- 
quantitative) * 

On the other hand^ most of the items that were ranked as 
being least influential involved feedback that came from stu-- 
dents* These included the students* test results and their 
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responses to multiple choice questionnaire items. Both of 
these procedures were types of feedback that could be cate- 
gorized as being non-verbal and formal {i>e>, quantitative) > 
This is not to say that feedback from students did not influ* 
ence these writers* minicourse revisions, but instead, that 
feedback from teachers had a greater influence than feedback 
from students* 

As is typical of most studies, an answer to a researchable 
question such as the one posed in this paper inevitably leads 
to additional questions > The results of this study lead one 
to wonder whether these revisors rated feedback from teachers 
to be more influential than feedback from students merely 
because it came from teachers rather than students^ or was it 
more influential because it was more verbal, less formal, and 
less quantitative than the student feedback data* The answer 
to this question, though not within the scope of this paper, 
is a logical 90al for further research.. 

One final note is in order here. The reader should be 
inade aware that the feedback procedures described in this 
paper were those that were followed in field testing ISIS 
trial-draft minicourses during the 1974-1975 school year.. 
Since that time feedback from revisors as well as changes in 
the content and format of minicourses have led to the modifi- 
cation of several of the feedback procedures described in this 
paper. Studies such as the one described in this paper have ' 
contributed information that aided our staff in deciding which 
feedback procedures could be revised. 

11 



APPENDIX A 
TO: All ISIS Minicourse Revlsors 
FROM: J. Clesla 

SUBJECT: How various sources of feedback affect your mlnlcour^:; 
revisions. 

In an attempt to help me do a study of the comparative Influence 
of the various kinds of feedback Information available to youf would 
you please respond anonymously to the following: 

Disregarding your own Intuitive feelings toward what sihould be 
revised In a given minicourse, and also disregarding the modifications 
that have been Imposed on you due to changes made In the objectives 
for a given minicourse, please rank In order of decreasing Importance 
the Influence that the following randomly ordered list of 10 types 
of feedback have upon your revisions of ISIS mlnlcourses: 

A) Suggestions made to me by the Project Director or Associate 
Directors. 

B) Reviews of the minicourse from ^content experts^. 

C) Audio taped comments from ISIS Trial Teachers. 

D) ISIS students* minicourse test results. 

E) Examining copies of minicour&e booklats used by students. 

F) ISIS students* responses to feedback questionnaire Items. 

G) Suggestions made to me by Projer.t staff members (excluding 
Evaluation Staff and Project A'Siviinlstrators) . 

H) Comments written by ISIS Teachers in minicourse booklets. 

I) My own visits to ISIS classrooms. 

J) ISIS Teacher;^* written responses to questionnaire items. 



Rank Order Letter; 

(Most Influential) 
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7 



(Least Influential) l0~ 
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